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ABSTRACT 

This study investigated whether National Board for 
Professional Teaching Standards (NBPTS) -certified teachers in Tennessee were 
exceptionally effective in bringing about objectively measured student 
achievement gains. Tennessee has over 40 NBPTS-certified teachers, 16 of whom 
teach in grades 3-8 and have value-added teacher reports in the state 
database. Tennessee's teacher reports summarize the annual achievement gains 
exhibited by each teachers' students. This study used teacher-effect scores 
from year 2000 teacher reports of the NBPTS-certified teachers included in 
the database. Data analysis indicated that the 16 teachers could not be 
considered exceptionally effective in terms of their ability to bring about 
student achievement . All of the teachers fell short of the standard either in 
one of the required subjects (mathematics, reading, and language) or failed 
to meet it for three consecutive years (where three years of data were 
available) . One teacher came close to qualifying, with exceptional teaching 
in two out of three subjects for two out of three years. An appendix presents 
teacher-effect scores for Tennessee NBPTS-certified teachers. (SM) 
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The Value-Added Achievement Gains of MBPTS-Certified Teachers in Tennessee: 

A Brief Report 

J. E. Stone, Ed.D. 

College of Education 
East Tennessee State University 

POLICYMAKER BRIEFING 

Controversy Re: Briefing & Study 



Introduction 

The National Board for Professional Teaching Standards (NBPTS) was established in 1987. Funded principally by the Carnegie 
Foundation, major teacher unions, and the U. S. Department of Education, its mission is to set advanced proficiency standards 
for teachers and to certify teachers who meet those standards. 

Teachers pay $2,300 to be evaluated by the NBPTS. They prepare a lengthy portfolio, make videotapes of themselves, and take 
an all-day written exam. Only fully licensed and experienced teachers may apply. 

States and school districts have offered powerful incentives, such as salary increases in the $5,000 to $7,500 per year range for 
successful applicants. As of January 2002, the NBPTS has certified 16,037 teachers in 19 areas of teaching. Over 20,000 more 
have applied. The NBPTS says certified teachers are "highly accomplished," and it anticipates that only 10% of all teachers will 
eventually be so designated. 

NBPTS standards are closely linked to the National Council for the Accreditation of Teacher Education's (NCATE) teacher 
training standards and the Interstate New Teacher Assessment and Support Consortium's (INTASC) teacher licensure 
standards. All are based on research, teacher opinion, and expert judgment; however, all reflect the same ideas about teaching 

that have dominated education for decades.- 

Several studies have attempted to show that NBPTS-certified teachers are truly superior in the classroom. The largest of these 
investigations-a study by the Center for Educational Research and Evaluation at the University of North 
Carolina-Greensboro-compared 31 teachers who were awarded certification to 34 teachers who applied but were 

p 

unsuccessful.- The results indicated that successful applicants were higher on 11 of the 13 qualities the NBPTS considers 
essential to good teaching. The student achievement outcome, however, was equivocal. 

Neither the UNCG study nor any others have affirmed that NBPTS certified teachers are exceptionally effective in improving 
objectively measured student achievement. Its assessment of student learning was based on work samples chosen by the 
participating teachers themselves-hardly an unbiased measure. Moreover, despite the key role of standardized tests in state 
accountability plans, the UNCG report argued that such tests are narrow, inadequate, and arbitrary. 

The purpose of the present report is to address whether NBPTS-certified teachers are exceptionally effective in bringing about 
objectively measured student achievement gains. 

Method 

Tennessee has more than 40 NBPTS certified teachers. Sixteen of the 40 teach in grades three through eight and therefore 
have value-added "teacher reports" in the state database. Tennessee's teacher reports summarize the annual achievement 

o 

gains exhibited by each teacher's students - Using the "mixed model" statistical methodology developed by Dr. William Sanders, 
student progress is estimated on the basis of how much students gain in comparison to their achievement increases in previous 
years. The Sanders estimates of teacher impact on learning-called "teacher-effect" scores--are considered the most fair and 

exact available.- 



The teacher-effect scores on which this report is based were furnished to the author by the Tennessee Department of 

Education.- They were extracted from the year 2000 teacher reports of the NBPTS-certified teachers who are included in the 
Tennessee Value Added Assessment System (TVAAS) database. Names and school system identifiers were not included so as 
to preserve teacher anonymity. 



The teacher-effect scores in this study are reported on a scale of zero to 50. They represent the estimated mean achievement 
gains of the students taught by each teacher, in each subject taught by that teacher. Although Tennessee tests its students 
annually with McGraw Hill's 'Terra Nova," not every teacher generates a teacher-effect score in every subject every year; thus 
some teachers may have data available for only one year and/or only in selected subjects. 



The central question to be answered by this report is whether Tennessee's NBPTS certified teachers are exceptionally 
successful in improving the achievement test scores of their students. For the purposes of this report, "exceptional" teaching is 
defined as teaching that brings about an improvement in student achievement equal to 115% of one year's academic growth in 
the local school system. 
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This definition is based on Tennessee's performance standards for schools and school systems. Annual gain in a given subject 
equaling or exceeding 1 1 5% of the national norm gain is considered "exemplary" and awarded a grade of "A." Not incidentally, a 
gain of less than 85% is considered "deficient" and awarded a grade of "F." 

A recently created Chattanooga, Tennessee program uses the 115% standard for identifying "high performing" teachers. 
Teachers whose students gain 115% of the local average in 3 core subjects earn a $5,000 bonus. 3 

Results 

The TVAAS teacher-effect scores for the 16 NBPTS-certified teachers are displayed in Appendix A of this report. The teachers 
(rows) are numbered 1-16, and each row contains a date, a grade level, and the teacher-effect scores for a given teacher. With 
the exception of the rows highlighted in blue (3-year averages), each row labeled "Teacher #" contains average student 
achievement gain scores for one year and in one or more of the five tested subjects: mathematics, reading language, social 
studies, and science. The rows labeled "System #" (immediately following each teacher row) contain the relevant school system 
average and standard error. 

Each teacher data cell contains the teacher-effect score followed by its standard error of measurement (in parenthesis) and by 
the percentage of annual achievement growth that it represents (in brackets). The percentage of annual growth is the critical 
indicator of teacher effectiveness in this study. It is the ratio of the teacher-effect score to the average annual achievement 
growth for the school system multiplied by 100. The percentage of average annual achievement growth is the teacher 
performance indicator used in the Chattanooga program. 

In the interest of enhancing the accessibility of this report, the use of statistics has been minimized. The reader is encouraged to 
inspect the data in Appendix A . The scores that would warrant a grade of "A" (i.e., scores that represent 115% or more average 
annual growth; and ones that would warrant a bonus in Chattanooga if achieved in mathematics, reading, and language) are 
highlighted in green. The scores that would warrant a grade of "F" (i.e., scores that represent 85% or less of annual growth) are 
highlighted in red. 

Even if one considers only the data available for each teacher, none of the 16 teachers would qualify for the bonus awarded 
teachers in the Chattanooga program. All fall short of the standard either in one of the required subjects (mathematics, reading, 
and language) or they fail to meet it for three consecutive years (where three years of data is available). Teacher 1 1 comes the 
closest to qualifying, with exceptional teaching in two out of the three subjects for two out of three years. 

Considering the 16 teachers collectively, there are 123 teacher-by-subject-by-year teacher-effect scores. Only 18 (15 percent) of 
these scores reach the "exemplary" or "A" level and 13(11 percent) would be designated as "deficient" and given a grade of "F." 

Discussion 

The 16 NBPTS certified teachers for whom TVAAS data is available cannot be considered exceptionally effective in terms of 
their ability to bring about student achievement. With the exception of the above noted highs and lows, the achievement gains 
made by their students are no greater than those made by students who had other teachers. None would have qualified for the 
bonus offered in Chattanooga. Plainly, these findings are distinctly at odds with that which policymakers and the public have 
been given to understand about the quality of NBPTS certified teachers. 

In an Education Week article titled "National Certification Found Valid for Teachers," NBPTS President Betty Castor said of the 
UNCG study: "It gives us-parents, elected officials, and policymakers-the absolute highest confidence that 

national-board-certified teachers are providing students with a high-quality learning experience." 7 Clearly her confidence was 
misplaced. 

Is there some chance that the teacher-effect scores earned by the teachers in this report are misleadingly low? 

Inevitably, some of the scores reported in any study are underestimates and some are overestimates. Annual estimates of the 
impact had by teachers on student achievement can be unstable but the errors in estimation tend to balance out when scores 

O 

are considered collectively 

As a practical gauge forjudging how many scores are needed to make a reasonable estimate of a given teacher's performance, 
consider the number of test scores that enter into a student grade for a semester-long college course. Assuming that a teacher 
in grades three through eight would have fifteen to twenty students, the estimated annual achievement gains for the teachers in 
this study are probably based on three or four times as many test scores as the typical college course grade. Stated a bit more 
technically, gain scores have a larger error component but TVAAS teacher-effect averages compensate by including a larger 
number of scores. 

Because Tennessee seeks to afford teachers a much higher degree of protection against inaccurate assessments than is 
typically assured by college grading practices, Tennessee’s use of "teacher effect" scores in annual teacher evaluations is 
statistically very conservative. An individual teacher is classified as above or below average only if his or her three-year rolling 
average is more than two standard errors above or below the system mean. 
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Is it necessary for a study of this kind to consider only three-year averages? 

Given that the precise classification of individual teachers is not the primary objective of the present analysis, the restrictions 
appropriate to clinical use of these data are not necessary. It should be noted, however, that none of the teachers for whom 
three years of data is available have an average of 1 15 percent, in any subject. Moreover, with the exception of the Social 
Studies scores of Teacher 1 , none of the teachers with less than three years of data would attain the 115 percent criterion 
without substantially improved second or third years. In other words, it is unlikely that any of the teachers in this study would 
have been classified as exceptional had 3 years of data been available. 

Are the 16 teachers in this study representative of all NBPTS-certified teachers? 

In any study, the available sample may misrepresent the population. In other words, it is at least statistically possible that while 
none of these NBPTS teachers appear to be exceptional, the sixteen thousand or so others for whom teacher-effect scores are 
unavailable would be so classified. 

How plausible is this conclusion? Not very. One would have to presume that the present sample is an anomaly-perhaps a group 
of teachers who test well but perform poorly in the classroom. In truth, it is more likely that the initial applicants for NBPTS 
certification would be above-average representatives of their group. 

Again, it is useful to consider the data from a practical standpoint: If a grocer bought several cases of premium, hand-wrapped, 
large apples and found the first box to be 85 percent mediums and smalls, he would certainly have good reason to be skeptical 
about the rest. Moreover, if the apples cost as much as NBPTS certified teachers, the grocer would be entirely justified in 
demanding a refund and finding a new supplier. 

There is one final point regarding the statistical limitations of this study: Most of the studies that have drawn favorable 
conclusions regarding NBPTS-certified teachers have had similar limitations. For example, the UNCG study discussed above 
included only 31 NBPTS-certified teachers. A 1995 study sponsored by the NBPTS compared three NBPTS-certified teachers to 

Q 

three non-certified teachers.- Plainly, a limited number of subjects was not considered a bar to scientific and practical 
importance in these earlier studies. 

Conclusion 

The findings of this study present a serious challenge to NBPTS's claims regarding its teacher quality standards and certification 
process. At the very least, they suggest that public expenditures on NBPTS certification and teacher bonuses should be 
suspended until it can be clearly and independently established that NBPTS certification delivers what it promises. 

The number of teachers in this study is small but comparable to the numbers of teachers in the studies on which the NBPTS has 
based its claims. Significantly, however, this study was independently conducted and it links NBPTS certification to the outcome 
of greatest interest to policymakers and the public. Although the findings are not definitive, they clearly indicate that the NBPTS 
standards and certification process-at least as presently constituted-are not serving the teacher quality aims of public policy. 

Are NBPTS certified teachers exceptional in some respect other than their ability to improve objectively measured student 
achievement? 

It may be that the teachers in this study are exemplars of the teaching practices idealized by NBPTS, NCATE, and INTASC and 
that their teacher-effect scores are only average because NBPTS, et al, treat measured student achievement as something less 
10 

than a top priority. — To the contrary, Tennessee and most other states treat measured achievement as an unrivaled priority; 
thus teachers who fail to produce exceptional student achievement would not be considered "exceptional" regardless of 
whatever else they are able to do. Contrary to the view expressed in the UNCG study, parents, policymakers, and the public 
regard gains in objectively measured student achievement as indispensable to good teaching. 

Can efforts to correct and revise the NBPTS initiative succeed? 

The discrepancy between the National Board's portrayal of its standards and the findings of this study may reveal critical flaw in 
the conceptual scheme on which NBPTS is founded. Medicine, law, and other professions have advanced certifications. 
Members of those professions, however, work in a marketplace where cost and quality considerations influence consumer 
behavior and revenue. By contrast, teachers work in a marketplace where the cost of added certificates, licenses, and 
credentials creates no competitive disadvantage even if they are meaningless. Under such conditions, the consumer's interest is 
virtually unprotected. 

May 1, 2002 
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Teacher-Effect Scores for TN NBPTS-Certified Teachers 

YearGrade Mathematics Reading Language Social Studies Science 



erIc 



Teacherl 


19996 




8.6 ( 2.0) [239] 


8.6 ( 2.1) [110] 


21.0 ( 2.8) [140] 


System 1 


19996 




3.6 ( 0.4) 


7.8 ( 0.4) 


15.0 (0.4) 


Teacherl 


20006 






5.2 ( 2.1) [82] 


24.0 ( 3.4) [364] 


System 1 


20006 • 






6.3 ( 0.4) 


6.6 ( 0.4) 


Teacher2 


19997 


16.2 ( 4.1) [109JL1J 








System2 


19997 


14.9(1.5) 








Teacher3 


20008 


13.3 (.4.1) [76] 








System3 


20008 


17.5 ( 1.6) 








Teacher4 


19996 


27.3(3.7) [100] 


6.9 (2.6) [98] 


8.2 ( : Z9) [ 76] . 




System4 


19996 


27.2 ( 1.6) 


7.0 ( 1.6) 


10.8 ( 1.6) 
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Teacher4 

System4 


20006 

20006 


26.9 ( 3.4) [106] 
25.4 ( 1.6) 


10.1 (2.8) [112] 14.0 (2.8) [91] 
9.0(15) 15.3(1.5) 


Teacher5 
System 5 


19986 

19986 




19.0 ( 2.4) [104] 14.0 ( 2.6) [113] 6 .0 ( 2.2) [ 79] Ql6.4 ( 2.3) [ 97] 
18.2(1.3) 12.4(1.3) 7.6(14) 16.9(1.5) 


Teacher6 

System6 


19987 

19987 




10.7 (2.5) [93] 15.1 (2.8) [98] 
11.5(2.5) 15.4(2.8) 


Teacher6 

System6 


19998 

19998 




14.1 (2.3) [107] 12.5(2.6) [102] 
13.2(2.6) 12.3(2.9) 


Teacher6 

System6 


20008 

20008 




14.4 (2.4) [100] 23.0 (2.7) [126] 

14.4(2.7) 18.3(3.0) 


Teacher7 

System7 


20004 

20004 


33.7 ( 3.9) [135] 
25.0 ( 0.8) 


136 ( 3.0) [115] 10.3 (2.7).[84]..1 3.7 ( 3.1 ) [1 09] 18.9 ( 2.9) [104] 
H. 8 (0,8) 12.3(0.8) 12.6(10) 18.1(0.9) 


Teacher7 

System7 


19985 

19985 


19.5 (2.8) [104] 
18.8(0.8) / 


13.1 (2.2) [116] 10.4 ( 2.7) [127] 1 5.8 ( 2.3) [1 03] 13.1 (2.4) [125] 
11.3(0.8) 8.2(0. 8) 15.3(0.9) 10.5 ( 0.9) ' 


Teacher7 

System7 


19995 

19995 


14.4 (3.2) [75] I 
19.2 (0!8) 


1 14 8 ( 2.2) [1 1 0] '1 0.2 ( 2.6) [1 1 0] 1 3.5 ( 2.5) [1 20] 1 9.4 ( 2.7) [1 00] 
13.4(0.9) " 9.3 (0.9) 11.2(0.9) 19.4( 0.9) 


Teacher8 

System8 


20004 

20004 


29.4(4.7) [106] 
27.6(1.6) 


10.0 ( 3.3) [ 93] 9.4 ( 319 ) [67]^14.7 ( 3.8) [100] 19.6 ( 3.9) [101] 
10.7(16) 14.1(15) 14.7(1.8) 19.5(1.7) 


Teacher9 

System9 


19987 

19987 


3.3(1.5) )r 


- 


Teacher9 

System9 


19997 

19997 


4.3 (2.2) [93] 
4.6 ( 1.6) 




Teacher9 

S'ystem9 


19998 

19998 


14.7(3.4) [104] 
14.2 (1.7) 




TeacherlO 19984 
System 10 19984 


28.4 (4.8) [110] 
25.8 (1.5) 


21.2 (2.7) [109] 21.9 (3.5) [113] 22.2 ( 2.9) [ 94] 25.9 ( 3.7) [100] 
19.4( 1.5) 19.3( 1.5) 23.7( 1.7) 25.9( 1.7) 


TeacherlO 19994 
System 10 19994 


34.8(3.6) [108] 
32.3 ( 1.6) 


14.3 (2.6) [103] 15.1 ( 3.2) [ 93] 21.3 ( 3.1) [102] 23.1 ( 3.3) [101] 
13.9( 1.6) 16.3(15) 20.9( 1.9) 22.8 ( 1.7) 


TeacherlO 19985 
. System 10 19985 


27.2 ( 4.6) [ 97] 
27.9 ( 1.5) 


15.2 (2.7) [98] 14.8 (3.4) [110] 21 .4 ( 2.9) [101] 16.7 ( 3.2) [103] 
15.5(1.5) 13.5(1.5) 21.2(1.6) 16.2(1.7) 



Year Grade Mathematics Reading Language Social Studies Science 



Teacherl 1 19987 


14.7 ( 2.4) [121]' 16.5 (2.3) [11 5] 




System 11 19987 


12.1(1.2) 14.4(14) 
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Teacherl 1 
System 1 1 


19997 

19997 


13.3 (2.3) [11 5] 10.6 (2.8) [138] 
11.6(1.3) 77(1.5) 


Teacherl 1 


20007 


12.3 (2.3) [103] ^2 (2.9) [84] 




System 1 1 


20007 


11.9(1.3) 7.3(15) 


Teacherl 1 


3vr 7 


13.4 ( 1.4) [1121 111 ( 1.6) [1131 


System 1 1 


3yr 7 


11.9(0.7) 9.8 (0.8) 



Teacher12 19987 


13.6 (1.8) [104] 




System 12 19987 


13.1 (0.5) 




Teacherl 2 19997 


|9.6 ( 1 .5) r 831 




System 12 19997 


11.6(0.5) 




Teacherl 2 20007 


7.8 ( 1.9) [111] 




System 1 2 2000 7 


7.0 ( 0.5) 




Teacherl 2 3yr 7 


10.3 ( 1.0) [981 




System 12 3yr 7 


10.5(0.3) 





Teacherl 3 


19986 


20.0 ( 1.7) [110] 


13.1 (2.3) [102] 


7.3 ( 1.7) [101] 


System 1 3 


19986 


18.2(0.8) 


12.9 (0.8) 


7.2 ( 0.9) 


Teacherl 3 


19996 


10.8 (2.2) [108] 


10.9 ( 3.1) [91] 


22.6 (2.0) [107] 


System 1 3 


19996 


10.0(0.9) 


12.0(0.9) 


21.2(0.9) 


Teacherl 3 


20006 


10.1 (3.5) [113] 


12.9 ( 3.0) [126] 


17.4(2.8) [105] 


System 13 


20006 


8.9 (0.8) 


10.2(0.8) 


16.6(0.8) 


Teacherl 3 


3yr 6 


13.6 ( 1.5) [1101 


12.3 ( 1.6) [1051 15.7 (1.3) [105] 


System 13 


3yr 6 


12.4(0.5) 


11.7(0.5) 


15.0 (0.5) 



Teacherl 4 


19985 


120.6 ( 3.0) [ 831 


13.7(2.4) [107] 


13.7(2.7) [120] 


19.6 ( 3.0) [98] 


13.0 (2.7) [98] 


System 1 4 


19985 


. 24.7 ( 1.4) 


12.8 ( 1.4) 


1 1 .4 ( 1 .4) 


19.9 ( 1.6) 


13.2 ( 1.6) 


Teacherl 4 


19995 


25.9(3.5) [114] 


11.7 (2.6) [87] 


10.4 (2.8) [98] 


9.9 (2.9) [91] 


16.1 (3.2) [119] 


System 1 4 


19995 


22.8 ( 1.4) 


13.4 ( 1.5) 


10.6 ( 1.5) 


10.9 ( 1.6) 


13.5 ( 1.5) 


Teacher14 


20005 


15.8 (4.8) [86] 


9.8 (3.1) [87] 


10.3 (3.5) [96] 


6.4 ( 3.3) [ 93] 


14.9(3.4) [119] 


System 1 4 


20005 


18.4 ( 1.5) 


11.3 ( 1.5) 


10.7 ( 1.6) 


6.9 (1.7) 


12.5 ( 1.6) 
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Each teacher effect score (cell) is accompanied by a standard error of measurement ( in parenthesis) and (in brackets) the percentage of annual 
achievement growth represented by the teacher effect score. The percentage of annual growth is the ratio of the teacher effect score to the 
average annual achievement growth for the school system (shown in the following row) times 100. 




9 BEST COPY AVAILABLE 



7 of 7 



01/03/2003 9:16 AM 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 




NOTICE 

Reproduction Basis 




This document is covered by a signed "Reproduction Release (Blanket)" 
form (on file within the ERIC system), encompassing all or classes of 
documents from its source organization and, therefore, does not require a 
"Specific Document" Release form. 



This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may be 
reproduced by ERIC without a signed Reproduction Release form (either 
"Specific Document" or "Blanket"). 






0 



EFF-089 (1/2003) 



